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The main objective of the program has been to implement and test the three-dimensional, 
block-structured Euler solver on computers with multiple processors. The developed 
scheme involves the partitioning of a large aerodynamics problem into several smaller 
problems where each represents a particular flow region. Each of these problems are 
Intel-IPSC computer with sixteen processors and IBM 3090 computer with four processors. 
In the case of IPSC, the memory is distributed between sixteen processors (4.5 megabytes 
each). In the case of IBM 3090, all four processors share the same large memory. 

Several test cases have been run on these computers and basic considerations for 
analyzing large problems on parallel computers have been investigated. 
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1. Objectives 


The main objective of the research program has been to develop and implement a 
three-dimensional, block-structured (zonal) solution scheme for Euler 
equations on computers with multiple processors. The methodology is aimed at 
developing computational techniques for solving three-dimensional, transonic 
flow problems around complex geometries. For solving problems such as 
transonic flow around an aircraft, one expects to employ many computers for 
the solution of a single problem. The developed scheme involves the 
partitioning of a large aerodynamics problem into several smaller problems 
where each sub-problem is defined for a particular flow region. Each of these 
problems are then solved on individual processors. The boundary condition 
between the neighboring flow regions are communicated between these processes. 
This work has been the continuation of a development over the years for 
generating grids for complex geometries and solving large problems for such 
geometries by using a zonal approach and finally studying the appropriate 
computer architectures for solving such problems. The main publications 
describing the research work are: 

1. Ecer, A., Spyropoulos, J.T., and Maul, J.D., "A Three-Dimensional, 
Block-Structured Finite Element Grid Generation Scheme," AIAA 
Journal, Vol. 23, Oct. 1985, pp. 1483-1490. 

2. Ecer, A. and Spyropoulos, J.T., "Block-Structured Solution Scheme for 
Analyzing Three-Dimensional Transonic Potential Flows," AIAA Journal, 
Vol. 25, Oct. 1987, pp. 1292-1300. 

Ecer, A., Spyropoulos, J.T. and Rubek, V., "Block-Structured Solution 
of Euler Equations for Transonic Flows," AIAA Journal, Vol. 25, Dec. 
1987, pp. 1570-1576. 
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4. Ecer, A., Spyropoulus, J.T., Sims, M. , Parallel Processing Techniques 
tor trie solution of Euler Equations, AIAA Paper 88-0620. AIAA 26th 
Aerospace Sciences Meeting, Reno, Nevada, January 11-14, 1988. 

2. Description of the Method of Solution 

The method of solution is documented in several papers in detail. Here a 
short summary of the methodology is provided: 

a) B1ock-Structured Grid Generation Scheme (ref. 1) . 

Flow field around a complex geometry is represented by an assembly of flow 
regions. Each region corresponds to a simpler geometry (a block) which can be 
defined easily. These blocks are attached to each other for describing the 
complex flow field. The block-structure can be irregular i.e. blocks can be 
attached to each other in an unstructured fashion. The procedure keeps track 
of the block-structure at all times. In designing a computational grid, one 
specifies the grid required for each block. The blocks are automatically 
connected to each other. Besides representing complex geometries, the blocks 
are used to model specific flow structures. For example, a block is placed to 
include the flow region where a shock occurs. One can then specify the 
required grid refinement around the shock and this grid is automatically 
connected to the neighboring flow regions. This methodology is also useful 
when one starts with a computational grid and modifies it locally after 
observing the accuracy of the obtained results. Any local grid modifications 
is propagated to the neighboring blocks automatically. 

b) Block-Structure Solution Schemes (ref 2,3) . 

After the computational grid is generated in terms of a series of blocks, the 
solution of Euler equation is defined in terms of the solution of the 
equations for each of the blocks with interface boundary conditions between 





the neighboring blocks. The steady-Euler equations are solved by using a 
relaxation scheme where the velocity vector is decomposed by using a Clebsch 
transformation as follows: 


U = 7$ + SVq 

where S is the enhopy, <j> and n are Lagrange multipliers. The Euler equations 
are written in terms of these variables as follows: 
v(pv<{>) = - v(pSVn) 

V. . puS) = 0 
PU . Vij = - p/R 

where p is the density, p is the pressure and R is the gas constant. For 
blocks with cut rctationbity (Constant entropy) one needs to solve only the 
first equation. The boundary conditions between the neighboring blocks 
requires the continuity of <j>, S and n- For potential equation, only the 
continuity of <j> is required. The solution scheme require iterative solution 
of these equations in each of the blocks together with the iteration of the 
boundary conditions between the neighboring blocks. An efficient methodology 
for iterating the boundary conditions for «f>, S and n have been developed, 
c) Parallel Processing for Block-Structured Solution of Euler 
Equations (ref.4): 

The developed methodology is well suited for parallel processing. For a given 
computer with many processors, the blocks are distributed among the 
processors. The information exchange between the neighboring processors 
involve only the exchange of boundary conditions which is a relatively small 
set of data. In terms of parallel processing, two types of computers were 
tested. For a computer with distributed memory (Intel-IPSC) each of the 
blocks is stored on an individual computer with its own memory. For a 
computer with shared memory (IBM 3090), the memory is shared by several 
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processors. In the first case, the numbers of processors and the blocks are 
matched. In the second case, the number of blocks is much larger than the 
number of processors. 

The objective of this part of the study was to investigate the applicability 
of parallel processing for solving large aerodynamics problems. The 
methodology is quite general and numerical results have been obtained for 
several practical problems in a parallel environment. The completed work 
resents one of the few applications of relaxation schemes in a parallel 
computing environment; 

3. Summary of the Research: 

The research work on the block-structured solution of Euler equations with 
parallel processing progressed along four main areas: 

A. Implementation of the Block-Structured Solution Scheme on a Intel 
hypercube with distributed memory and sixteen processors. 

B. Implementation of the Block-Structured Solution Scheme on a IBM 3090 
computer with shared memory and four processors. 

C. Configuration of Grid Generation Studies. 

A. Implementation of the Block-Structured Solution Scheme on a Intel 
Hypercube. 

This activity involved the solution of aerodynamics problems on a 
computer with many processors each with its own individual memory (distributed 
memory). The example in figure 1 shows the sixteen block model of a wing body 
problem. In this case, each block is stored on an separate computer. Grids 
are generated on a block-by-block bases and the solution is obtained for each 
block. The only exchange of information between the neighboring blocks is the 




















boundary conditions. This application is one of the first implementations of 
a three-dimensional CFD code on a Intel Hypercube computer. 


The following papers were written describing this work: 

1. Ecer, A., Spyropoulus, J.T., Sims, M., Parallel Processing Techniques 
for the solution of Euler Equations, AIAA Paper 88-0620. AIAA 26th 
Aerospace Sciences Meeting, Reno, Nevada, January 11-14, 1988. 

2. Ecer, A., Spyropoulus, J.T., Block Structured Solution of Euler and 
Naiver-Stokes Equations, 2nd Int. Symp. on Domain Compositions 
Methods, USC^ Los Angeles, January 14-16, 1988. (invited paper) 

3. Ecer, A., Spyropoulus, J.T., and Sims, M., The Solution of Euler 
Equations on a Hypercube, 3rd Conf. on Hypercube Concurrent Computers 
and App., Caltech, Pasadena, January 19-20, 1988. 

4. Ecer. A. and Akay, H.U., A Block-Structured Finite Element Solution 
of Viscous Internal Flows, Int. Conf. on Comp. Meth. in Flow 
Analysis, September 4-8, 1988, Okayoma, Japan, (invited paper) 

5. Ecer, A., & Spyropoulos, J.T., "Parallel Processing Schemes for the 
Block-Structured Solution of Transonic Flows," Society Computer 
Simulation Summer Conference, Ontario, Canada, July 23-29, 1987. 

B. Implementation of the Block-Structured Solution Scheme on IBM-3090 
Computer. 

In this case, the same problem was implemented on a computer where four 
processors share the same large memory as shown in figure 2. Many blocks are 
solved on each of the processors where one can address several gigabytes of 
memory in an efficient manner by using the developed scheme. Presently a 12 
processor version of this application is attempted. We have been 
collaborating with IBM research center in Kingston on this activity. 








Block-structured solution scheme on an IBM 3090-E 
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The implementation of the scheme on parallel processors for solving large 
problems was investigated by using the developed capability. Numerical 
results were obtained for a nacelle and a wing-nacelle configuration which 
involve over 150 blocks as shown in figure 3. Presently a F-16 geometry 
aircraft (over 500,000 grid points) is under investigation as shown in figure 
4. 

The following papers was prepared describing this activity: 

1. Chang, S.M., Ecer, A. and Spyropoulos, J.T., "The Use of IBM Parallel 
FORTRAN and Multitasking facility with a Domain Decomposition 
Approach for the Euler Equations," IBM Symposium on Parallel 
Processing Poughkeepsie, New York, Oct. 19-21, 1988. 

2. Ecer, A., Spyropoulos, J.T. & Chang, S.M. "Parallel Processing 
Schemes for the Block-Structured Solution of Transonic Flows II," 
Society of Computer Simulation Summer Conference, Seattle, 

Washington, July 25-28, 1988. 

3. Chang, S.M., Ecer, A. and Spyropoulos, J.T., "Parallel Processing 
Techniques of the Euler Equations on the IBM 3090 VF Computer," The 
Second International Conference on Vector & Parallel Computing Issues 
in Applied Research and Development, Tromso, Norway, June 6-10, 1988. 

4. Chang, S.M., Ecer, A. and Spyropoulos, J.T., "Parallel Processing 
Techniques on the IBM 3090VF Computer System," 7th International 
Conference on Finite Element Methods in Flow Problems, Huntsville, 
Alabama, Apr. 3-7, 1989. 

5. Spyropoulos, J.T., Ecer, A. Rout, R.K., and Badesha, S.S., "Block- 
Structured Solution of Three-Dimensional Transonic Flowfields for 
Wing/Pylon/Nacelle configurations," AIAA Paper 89-2940, 25th Joint 
Propulsion Conference, Monterey, California, July 10-14, 1989. 
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Jrid for the F-16 Aircraft 
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C. Grid Generation Studies. 

The continuation of the grid generation studies were basically focused on 
testing the accuracy and efficiency of the scheme for complex geometries. 

The following papers were written describing this activity: 

1. Ecer, A., Spyropoulos, J.T., Application of a Three-Dimensional 
Finite Element Grid Equation Scheme for Analyzing Transonic Flow 
Around an Aircraft, Conf. on Automated mesh Generation and 
Adaptation, Grenoble, France, Oct. 1987. (invited) 

2. Ecer, A., Spyropoulos, O.T. and Bulbul E., Application of a Three- 
Dimensional Finite Element Grid Generation Scheme for F-16 Aircraft 
Configuration, 2nd Int. Conf. on Num Grid. Generation in Comp. Fluid 
Dynamics, Miami Beach, Dec. 5-8, 1988. 

3. Ecer, A., Akay, H.U. and Spyropoulos, J.T., "Applications of a Three- 
Dimensional Finite Element Grid Generation Scheme to Flow Problems," 
5th International Conference on Numerical Methods in Laminar and 
Turbulent Flows, Montreal, Canada, July 6-10, 1987. 

4. Conclusions 

The main objective of the research has been the development of methodology for 
solving large transonic flow problems with complex geometries. One can 
develop a basic computational procedure for solving a particular flow problem 
around a two-dimensional simple geometry (eq. an airfoil). However, it is not 
always easily possible to perject the applicability of this procedure for 
solving, complex, three-dimensional flow problems. We have been using finite 
element methods for solving CFD problems basically due to their flexibility 
in modeling complex geometries (unstructured grids). However, our experience 
with large problems indicated that it is very difficult to treat large 
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problems as a single problem. Even if one can generate "a grid" for a complex 
geometry it requires several iterations to ensure that this is "an acceptable 
grid" in terms of providing disired accuracy with available computational 
resources. In terms of solving Euler equations, we have also reduced the 
problem to the solution of potential or non-isentropic potential equation for 
the majority of the flow field with the solution of the Euler equations only 
at certain critical regions. In terms of accuracy of the solutions in 
isentropic flow regions, the solution of potential equations are quite 
satisfactory with coatser grids then the ones used for Euler equations 
docummented in the literature. In the developed block-structured solution 
scheme, one can define which specific regions (blocks) will be analyzed by 

using the Euler equations and provide desired grid accuracy in these regions. 

The block-structure solution schemes become even more powerful when the 
solution of Naiver-Stokes equations are sought which requires considerably 
refined grids. 

The parallel processing of the Euler equations proved to be a natural 
extension of the developed technology. While the computational speed and size 
of computers is expected to increase in coming years, one can easily expect 

8 9 

grids with 10 - 10 grid points to be still quite expensive on a single 

processor. There is agreement among many researchers that parallel processing 
may be the most feasible way for solving such large problems. It is important 

to note that if one is going to solve a large problem on parallel computers 

one should consider the methodology which will work efficiently on such 
machines. As can be seen from the present research, the grid generation, the 
solution scheme and the computer implementation must be considered as a single 
problem. The conversion of an existing computational procedure and a computer 
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code to a parallel machine does not always produce impressive results. Some 
of the current issues in CFD such as adaptive grids, efficiency and accuracy 
of computational schemes has to be studied in conduction their with 
application to parallel computers. For example, many computational schemes 
which have been developed for structured grids are being revised in terms of 
their application to unstructured grids. One expects a similar activity in 
the future, in terms of implementing dif f erent schemes for parallel computers. 
Instead, it seems more logical to develop computational schemes with the 
objective that they w-ill work on parallel computers. The completed work 
provides such an example where a complete methodology is developed for solving 
CFD problems, in particular transonic flows, in a parallel computing 
enviroment. 







